feat: cross-platform CI matrix and runtime sandbox hardening - #535

ares_process_fd can call back into sock_state_cb ->
spine_async_dns_upsert_watcher -> spine_async_dns_remove_watcher,
which frees the current watcher mid-call. Any dereference of
`watcher` after ares_process_fd is UAF. The current code happens to
read fields only before the call, but there is no comment documenting
that invariant and a future refactor would break it silently.

Capture the fd explicitly before the call, document that runtime is
the only pointer safe to use post-call, and make spine_async_dns_on_poll
self-contained. No functional change.

Signed-off-by: Thomas Vincent <thomasvincent@gmail.com>

Before uv_loop_close, spine runs db_disconnect on every MYSQL handle.
An async query that arrives between the flush drain and the db close
would succeed at submission, then fire its callback after the handle
was freed - use-after-free.

Add a process-wide shutdown fence:

- spine_async_mysql_shutdown_begin() flips a volatile flag; new
  queries via spine_async_mysql_query return -ESHUTDOWN.
- In-flight queries keep their existing uv_poll chain so the uv_run
  drain completes them normally.
- Non-HAVE_MYSQL_ASYNC build provides the symbol as a no-op so
  callers in spine.c do not need the compile-time guard.

Spine calls the fence immediately before the batch flush so the
flush's own callbacks still run, and then no further submissions are
possible while the drain is in progress.

Signed-off-by: Thomas Vincent <thomasvincent@gmail.com>

M2: setrlimit(RLIMIT_CORE, 0) silently ignored its return. On systems
with /etc/security/limits.conf enforcing a non-zero hard minimum, it
left the limit unchanged and the operator had no signal. Log to
stderr on failure so post-mortem cores have a documented cause.

M6: Dependabot weekly cadence for docker left a 7-day exposure
window on base-image CVEs. Switch to daily for docker; github-actions
stays weekly because action SHAs churn slowly and noisy.

Signed-off-by: Thomas Vincent <thomasvincent@gmail.com>

Two #define SPINE_BZ / #undef pairs existed — one per scrub helper —
each selecting explicit_bzero vs spine_volatile_bzero based on
HAVE_EXPLICIT_BZERO. Collapse them into one static inline spine_bzero
with a single #ifdef. Every scrub call routes through it; adding a
new scrubbable field no longer requires remembering which macro
pair a file is using.

Also extend target_t scrub to cover snmp_engine_id and snmp_context.
They are not strictly secret, but they are per-device identifiers
operators prefer not to surface in a core dump alongside a password.

Signed-off-by: Thomas Vincent <thomasvincent@gmail.com>

The prior ordering flipped the async-mysql shutdown fence BEFORE
calling spine_async_batch_flush. spine_async_batch_flush submits
pending batched writes through spine_async_mysql_query; with the
fence already set, those submissions returned -ESHUTDOWN and the
entire pending batch was silently dropped - the same data-loss
mode the dummy-flush removal was supposed to fix.

Swap the order: flush first, drain callbacks, then close the gate.

Signed-off-by: Thomas Vincent <thomasvincent@gmail.com>

The prior flag list included -3p / -3x / --3authPassphrase /
--3privPassphrase. Those are not real net-snmp options; they were
added based on assumption, not documentation. Tests asserted
redaction against strings that never appear in real invocations.

Replace with the actual net-snmp key-carrying short flags from
snmp.conf(5):
  -3m  master auth key
  -3M  master priv key
  -3k  localized auth key
  -3K  localized priv key

Drop the fabricated long flags. Update tests to exercise the real
flag forms.

Signed-off-by: Thomas Vincent <thomasvincent@gmail.com>

Prior code used UV_RUN_NOWAIT in a 300-iteration spin, which returned
immediately regardless of outstanding requests. The 3-second deadline
was fiction: 300 iterations completed in microseconds, not seconds,
and the runtime was freed while requests were still in flight. The
next on_resolved dereferenced ctx->runtime (now freed) to decrement
inflight - UAF.

Switch to UV_RUN_ONCE driven by a wall-clock deadline (time(NULL)+3).
UV_RUN_ONCE blocks until one libuv event fires or the loop has no
pending work, so the 3-second budget is actually spent waiting.

If the deadline still expires with requests in flight, DO NOT free
runtime. The outstanding callbacks carry a pointer to it; freeing
now is UAF. Leak the struct (process is terminating) and log a
warning so operators can see the stuck resolver.

Signed-off-by: Thomas Vincent <thomasvincent@gmail.com>

Three issues flagged in self-review:

1. The reap pass only fired from spine_cb_record. On an idle daemon
   (no poll failures), the reap never ran and old entries from a
   prior failure burst sat in the hash for the process lifetime.
   Add the same reap call to spine_cb_should_skip, which is invoked
   every cycle for every host. Rate limiting via
   SPINE_CB_REAP_INTERVAL already prevents thrash.

2. The reap sweep held the table mutex O(N). A large unhealthy
   population stalled other threads. Add SPINE_CB_REAP_BATCH (256)
   cap; unscanned entries wait for the next sweep.

3. Tests could not drive the reap deterministically because it used
   time(2). Add spine_cb_set_clock_for_test(fn) and thread the clock
   through every time(NULL) caller via spine_cb_clock().

Signed-off-by: Thomas Vincent <thomasvincent@gmail.com>

Two self-review findings:

- g_spine_uv_leaked_handles was a plain int. Single-threaded today,
  but a future refactor that walks the loop from a worker thread
  would race on increment. Switch to _Atomic int with relaxed-order
  fetch_add; this is only an operator diagnostic so no synchronisation
  guarantees are required beyond 'no lost increments'.

- g_async_mysql_shutting_down was a plain volatile int. Worker
  threads on weakly-ordered architectures (ARM, POWER) might observe
  stale 0 after the main thread sets it. Switch to atomic_int with
  explicit release on the set and acquire on the read so the
  happens-before ordering is enforced across thread boundaries.

Signed-off-by: Thomas Vincent <thomasvincent@gmail.com>